Online learning enhanced atlas-based auto-segmentation

ABSTRACT

An image segmentation method is disclosed. The method includes receiving a plurality of atlases and a subject image, each atlas including an atlas image showing a structure of interest and associated structure delineations, the subject image being acquired by an image acquisition device and showing the structure of interest. The method further includes calculating, by an image processor, mapped atlases by registering the respective atlases to the subject image, and determining, by the image processor, a first structure label map for the subject image based on the mapped atlases. The method also includes training, by the image processor, a structure classifier using a subset of the mapped atlases, and determining, by the image processor, a second structure label map for the subject image by applying the trained structure classifier to one or more subject image points in the subject image. The method additional includes combining, by the image processor, the first label map and the second label map to generate a third label map representative of the structure of interest.

TECHNOLOGY FIELD

The disclosure relates to atlas-based auto-segmentation (ABAS) and, moreparticularly, to systems and methods for online learning enhanced ABAS.

BACKGROUND

Determination of structures, such as organs, in a patient and extractionof the shape of a structure is essential to many medical imagingapplications, such as diagnostic imaging, image-guided surgery, orimage-guided radiation therapy. In such applications, a target structureor target structures need to be determined from an image, such as acomputed tomography (CT) image, of the patient. The determination of thetarget structure(s) in the patient is usually known as structurecontouring or segmentation. Although manual contouring by human experts,also referred to as raters, is still a common approach for high qualitysegmentation in clinics, manual contouring is tedious andtime-consuming, and may suffer from large intra- and/or inter-ratervariations.

Automated segmentation of images can be challenging due to noises andother artifacts, as well as limited image contrast for many soft-tissuestructures. In recent years, atlas-based auto-segmentation (ABAS)techniques have shown promise as a solution. The ABAS includesperforming segmentation of a subject image using one or morepreviously-segmented images, such as segmented images from previouslytreated patients or from previous treatments of the same subjectpatient. The previously-segmented images together with theirannotations, e.g., structure label maps or structure surfaces, arereferred to as atlases. By aligning an atlas image to a new subjectimage obtained from the subject patient through image matching, alsoreferred to as image registration, an image transformation is computed.Structure labels for the subject image are produced by mapping structurelabels defined on the atlas to the subject image using the computedimage transformation.

The accuracy of ABAS usually relies on the quality and quantity of atlasimages used. For example, multiple atlases can be used during the ABASprocess to provide redundancy. On the other hand, atlas images showingsimilar underlying objects of those in the subject image may also helpimprove accuracy in labeling the subject image. The disclosed methodsand systems are designed to further improve the accuracy of conventionalABAS for image segmentation.

SUMMARY

One aspect of the disclosure is directed to an image segmentationmethod. The method includes receiving a plurality of atlases and asubject image, each atlas including an atlas image showing a structureof interest and associated structure delineations, the subject imagebeing acquired by an image acquisition device and showing the structureof interest. The method further includes calculating, by an imageprocessor, mapped atlases by registering the respective atlases to thesubject image, and determining, by the image processor, a firststructure label map for the subject image based on the mapped atlases.The method also includes training, by the image processor, a structureclassifier using a subset of the mapped atlases, and determining, by theimage processor, a second structure label map for the subject image byapplying the trained structure classifier to one or more subject imagepoints in the subject image. The method additional includes combining,by the image processor, the first label map and the second label map togenerate a third label map representative of the structure of interest.

Another aspect of the disclosure is directed to an image segmentationsystem. The system includes a memory and an image processor coupled tothe memory. The memory is configured to receive and store a plurality ofatlases and a subject image, each atlas including an atlas image showinga structure of interest and associated structure delineations, thesubject image being acquired by an image acquisition device and showingthe structure of interest. The image processor is configured tocalculate mapped atlases by registering the respective atlases to thesubject image, and determine a first structure label map for the subjectimage based on the mapped atlases. The image processor is furtherconfigured to train a structure classifier using a subset of the mappedatlases, and determine a second structure label map for the subjectimage by applying the trained structure classifier to one or moresubject image points in the subject image. The image processor is alsoconfigured to combine the first label map and the second label map togenerate a third label map representative of the structure of interest.

Yet another aspect of the disclosure is directed to a non-transitorycomputer-readable storage medium storing instructions that, whenexecuted by an image processor, cause the processor to perform an imagesegmentation method. The method includes receiving a plurality ofatlases and a subject image, each atlas including an atlas image showinga structure of interest and associated structure delineations, thesubject image being acquired by an image acquisition device and showingthe structure of interest. The method further includes calculatingmapped atlases by registering the respective atlases to the subjectimage, and determining a first structure label map for the subject imagebased on the mapped atlases. The method also includes training astructure classifier using a subset of the mapped atlases, anddetermining a second structure label map for the subject image byapplying the trained structure classifier to one or more subject imagepoints in the subject image. The method additional includes combiningthe first label map and the second label map to generate a third labelmap representative of the structure of interest.

Features and advantages consistent with the disclosure will be set forthin part in the description which follows, and in part will be obviousfrom the description, or may be learned by practice of the disclosure.Such features and advantages will be realized and attained by means ofthe elements and combinations particularly pointed out in the appendedclaims.

It is to be understood that both the foregoing general description andthe following detailed description are exemplary and explanatory onlyand are not restrictive of the invention, as claimed.

BRIEF DESCRIPTION OF DRAWING

In the drawings, which are not necessarily drawn to scale, like numeralsmay describe similar components in different views. Like numerals havingdifferent letter suffixes may represent different instances of similarcomponents. The drawings illustrate generally, by way of example, butnot by way of limitation, various embodiments discussed in the presentdocument.

FIG. 1 is a block diagram showing an exemplary radiotherapy system,according to some embodiments of the present disclosure.

FIG. 2 depicts an exemplary image-guided radiotherapy device, accordingto some embodiments of the present disclosure.

FIG. 3 depicts an exemplary image segmentation system, according to someembodiments of the present disclosure.

FIG. 4 is a flow chart illustrating an exemplary image segmentationmethod, performed by the image segmentation system of FIG. 3, accordingto an exemplary embodiment.

FIG. 5 is a flow chart illustrating an exemplary structure classifiertraining method, performed by the image segmentation system of FIG. 3,according to an exemplary embodiment.

FIGS. 6A and 6B show comparisons between segmentation results from aconventional method and an exemplary method consistent with the presentdisclosure.

DETAILED DESCRIPTION

Hereinafter, embodiments consistent with the disclosure will bedescribed with reference to the drawings. Wherever possible, the samereference numbers will be used throughout the drawings to refer to thesame or like parts.

Embodiments consistent with the disclosure include systems and methodsfor online learning enhanced atlas-based auto-segmentation (ABAS).Detailed descriptions of some examples not specified herein, such asdetailed descriptions of the ABAS method, can be found in Applicant'sprior application, issued as U.S. Pat. No. 9,122,950, the entirecontents of which are incorporated herein by reference.

The disclosed systems and methods are generally designed to segment asubject image based on one or more atlases. As used consistently herein,an “atlas” includes an image and corresponding structure delineations(annotations) indicating what structure(s) the image points belong to. A“subject image” is an image of a subject patient and is yet to besegmented. The disclosed methods and systems will be applied to segmentthe subject image based on one or more atlases. The subject image can beacquired by image acquisition devices.

The image in an atlas, also referred to as an atlas image, can be animage of another patient or a previous image of the subject patienttaken at an earlier time. In the disclosure, an image, either thesubject image or an atlas image, includes a plurality of image points,which can be referred to as pixels if the image is a two-dimensional(2D) image or voxels if the image is a three-dimensional (3D) image. Animage point in the subject image is referred to as a subject imagepoint. Similarly, an image point in an atlas image is referred to as anatlas image point.

The structure delineations can be represented as, for example, structurelabel maps, structure surfaces, or structure contours. The descriptionbelow uses the label maps as an example of the structure delineationsand is similarly applied to the scenarios of structure surfaces andcontours. A label map refers to a map of structure labels eachidentifying a corresponding image point as being within a particularstructure of interest. Alternatively, consistent with this disclosure, alabel map may also be a probability map, which contains structure labelsthat each represents the probability of the image point belonging to thestructure. For example, when segmenting a subject image includingmultiple structures, a structure label of an image point may provide aset of probability values indicting how likely the image point belongingto each of the structures under consideration. The disclosed systems andmethods provide an estimated structure label map for the subject image.

FIG. 1 is a block diagram showing an exemplary radiotherapy system 100,according to some embodiments of the present disclosure. Radiotherapysystem 100 may be an IGRT system. As shown in FIG. 1, radiotherapysystem 100 may include a control console 110, a database 120, aradiotherapy device 130, and an image acquisition device 140. In someembodiments, radiotherapy device 130 and image acquisition device 140may be integrated into a single image-guided radiotherapy device 150, asindicated by the dashed box 150 in FIG. 1. In some embodiments,radiotherapy device 130 and image acquisition device 140 may be separatedevices. In some embodiments, radiotherapy device 130 and imageacquisition device 140 may be physically or communicative connected toeach other, as indicated by a dotted-dashed line between radiotherapydevice 130 and image acquisition device 140 in FIG. 1.

Control console 110 may include hardware and software components tocontrol radiotherapy device 130 and image acquisition device 140 and/orto perform functions or operations such as treatment planning, treatmentexecution, image acquisition, image processing, motion tracking, motionmanagement, or other tasks involved in a radiotherapy process. Thehardware components of control console 110 may include one or morecomputers (e.g., general purpose computers, workstations, servers,terminals, portable/mobile devices, etc.); processor devices (e.g.,central processing units (CPUs), graphics processing units (GPUs),microprocessors, digital signal processors (DSPs), field programmablegate arrays (FPGAs), special-purpose or specially-designed processors,etc.); memory/storage devices (e.g., read-only memories (ROMs), randomaccess memories (RAMs), flash memories, hard drives, optical disks,solid-state drives (SSDs), etc.); input devices (e.g., keyboards, mice,touch screens, mics, buttons, knobs, trackballs, levers, handles,joysticks, etc.); output devices (e.g., displays, printers, speakers,vibration devices, etc.); or other suitable hardware. The softwarecomponents of control console 110 may include operation system software,application software, etc. For example, as shown in FIG. 1, controlconsole 110 may include treatment planning/delivery software 115 thatmay be stored in a memory/storage device of control console 110.Software 115 may include computer readable and executable codes orinstructions for performing the processes described in detail below. Forexample, a processor device of control console 110 may becommunicatively connected to a memory/storage device storing software115 to access and execute the codes or instructions. The execution ofthe codes or instructions may cause the processor device to performoperations to achieve one or more functions consistent with thedisclosed embodiments.

Control console 110 may be communicatively connected to database 120 toaccess data. In some embodiments, database 120 may be implemented usinglocal hardware devices, such as one or more hard drives, optical disks,and/or servers that are in the proximity of control console 110. In someembodiments, database 120 may be implemented in a data center or aserver located remotely with respect to control console 110. Controlconsole 110 may access data stored in database 120 through wired orwireless communication.

Database 120 may include patient data 122. Patient data may includeinformation such as (1) imaging data associated with a patientanatomical region, organ, or volume of interest segmentation data (e.g.,MRI, CT, X-ray, PET, SPECT, and the like); (2) functional organ modelingdata (e.g., serial versus parallel organs, and appropriate dose responsemodels); (3) radiation dosage data (e.g., may include dose-volumehistogram (DVH) information); or (4) other clinical information aboutthe patient or course of treatment.

Database 120 may include machine data 124. Machine data 124 may includeinformation associated with radiotherapy device 130, image acquisitiondevice 140, or other machines relevant to radiotherapy, such asradiation beam size, arc placement, on/off time duration, radiationtreatment plan data, multi-leaf collimator (MLC) configuration, MRIpulse sequence, and the like.

Image acquisition device 140 may provide medical images of a patient.For example, image acquisition device 140 may provide one or more of MRIimages (e.g., 2D MRI, 3D MRI, 2D streaming MRI, 4D volumetric MRI, 4Dcine MRI); Computed Tomography (CT) images; Cone-Beam CT images;Positron Emission Tomography (PET) images; functional MRI images (e.g.,fMRI, DCE-MRI, diffusion MRI); X-ray images; fluoroscopic images;ultrasound images; radiotherapy portal images; Single-Photo EmissionComputed Tomography (SPECT) images; and the like. Accordingly, imageacquisition device 140 may include an MRI imaging device, a CT imagingdevice, a PET imaging device, an ultrasound imaging device, afluoroscopic device, a SPECT imaging device, or other medical imagingdevices for obtaining the medical images of the patient.

Radiotherapy device 130 may include a Leksell Gamma Knife, a linearaccelerator or LINAC, or other suitable devices capable of deliveringradiation to an anatomical region of interest of a patient in acontrollable manner.

FIG. 2 depicts an exemplary image-guided radiotherapy system 200,consistent with disclosed embodiments. As shown, system 200 may includea couch 210, an image acquisition device 220, and a radiation deliverydevice 230. System 200 delivers radiation therapy to a patient inaccordance with a radiotherapy treatment plan. In some embodiments,image acquisition device 220 may correspond to image acquisition device140 in FIG. 1 that may acquire a subject patient image.

Couch 210 may support a patient (not shown) during a treatment session.In some implementations, couch 210 may move along a horizontal,translation axis (labelled “I”), such that couch 210 can move thepatient resting on couch 210 into and/or out of system 200. Couch 210may also rotate around a central vertical axis of rotation, transverseto the translation axis. To allow such movement or rotation, couch 210may have motors (not shown) enabling the couch to move in variousdirections and to rotate along various axes. A controller (not shown)may control these movements or rotations in order to properly positionthe patient according to a treatment plan.

In some embodiments, image acquisition device 220 may include an MRImachine used to acquire 2D or 3D MRI images of the patient before,during, and/or after a treatment session. Image acquisition device 220may include a magnet 221 for generating a primary magnetic field formagnetic resonance imaging. The magnetic field lines generated byoperation of magnet 221 may run substantially parallel to the centraltranslation axis I. Magnet 221 may include one or more coils with anaxis that runs parallel to the translation axis I. In some embodiments,the one or more coils in magnet 221 may be spaced such that a centralwindow 223 of magnet 221 is free of coils. In other embodiments, thecoils in magnet 221 may be thin enough or of a reduced density such thatthey are substantially transparent to radiation of the wavelengthgenerated by radiotherapy device 230. Image acquisition device 320 mayalso include one or more shielding coils, which may generate a magneticfield outside magnet 221 of approximately equal magnitude and oppositepolarity in order to cancel or reduce any magnetic field outside ofmagnet 221. As described below, radiation source 231 of radiotherapydevice 230 may be positioned in the region where the magnetic field iscancelled, at least to a first order, or reduced.

Image acquisition device 220 may also include two gradient coils 225 and226, which may generate a gradient magnetic field that is superposed onthe primary magnetic field. Coils 225 and 226 may generate a gradient inthe resultant magnetic field that allows spatial encoding of the protonsso that their position can be determined. Gradient coils 225 and 226 maybe positioned around a common central axis with the magnet 221, and maybe displaced along that central axis. The displacement may create a gap,or window, between coils 225 and 226. In the embodiments wherein magnet221 also includes a central window 223 between coils, the two windowsmay be aligned with each other. In some embodiments, image acquisitiondevice 320 may be an imaging device other than an MRI, such as an X-ray,a CT, a CBCT, a spiral CT, a PET, a SPECT, an optical tomography, afluorescence imaging, ultrasound imaging, or radiotherapy portal imagingdevice, etc.

Radiotherapy device 230 may include the source of radiation 231, such asan X-ray source or a linear accelerator, and a multi-leaf collimator(MLC) 233. Radiotherapy device 230 may be mounted on a chassis 235. Oneor more chassis motors (not shown) may rotate chassis 235 around couch210 when couch 210 is inserted into the treatment area. In anembodiment, chassis 235 may be continuously rotatable around couch 210,when couch 210 is inserted into the treatment area. Chassis 235 may alsohave an attached radiation detector (not shown), preferably locatedopposite to radiation source 231 and with the rotational axis of chassis335 positioned between radiation source 231 and the detector. Further,device 230 may include control circuitry (not shown) used to control,for example, one or more of couch 210, image acquisition device 220, andradiotherapy device 230. The control circuitry of radiotherapy device230 may be integrated within system 200 or remote from it, and isfunctionally represented by control console 110 shown in FIG. 1.

During a radiotherapy treatment session, a patient may be positioned oncouch 210. System 200 may then move couch 310 into the treatment areadefined by magnetic coils 221, 225, 226, and chassis 235. Controlconsole 240 may then control radiation source 231, MLC 233, and thechassis motor(s) to deliver radiation to the patient through the windowbetween coils 225 and 226 according to a radiotherapy treatment plan.

FIG. 3 depicts an exemplary image segmentation system 300, consistentwith disclosed embodiments. In some embodiments, image segmentationsystem 300 may include medical image processing device 310 and imagedatabase 320, and may be connected to image acquisition device 140 (or320) through network 360.

Database 320 may be configured to store one or more subject images and aplurality of atlas images and corresponding structure delineations. Thesubject images and atlas images may be either 2D or 3D images. In someembodiments, database 320 may be part of an oncology information systemthat manages oncology treatment plans for patients. In some aspects,database 320 may receive these image sets from an image database havingimages previously acquired during one or more radiotherapy treatmentsessions. In some embodiments, image acquisition device 140 may acquirethe subject images and store them in image database 320. The imagesstored in image database 320 may also correspond to images acquiredduring one or more radiotherapy treatment sessions.

In some aspects, medical image processing device 310 may be configuredto segment the subject images based on the atlas images and theirdelineations. In one embodiment, medical image processing device 310 maybe integrated into control console 110 or radiotherapy device 130, shownin FIG. 1. medical image processing device 310 may output segmentationresults to treatment planning/delivery software 115 to assist theplanning of radiotherapy treatment. Control console 110 may controlradiotherapy device 130 to direct a therapy, such as radiation beams, tothe structure of interest of the subject patient according to the dataof the segmentation results.

In some embodiments, medical image processing device 310 may include animage processor 311, a memory 312, an input/output interface 313, anetwork interface 314, a display 315, and a storage device 316.Components of medical image processing device 310 may be connected via aBUS. Medical image processing device 310 may be implemented using aspecial-purpose computer, or a general-purpose computer. For example,medical image processing device 310 may be implemented using a computercustom-built for hospitals to perform image acquisition and imageprocessing tasks.

In some embodiments, image processor 311 may be one or moregeneral-purpose processing devices, such as a microprocessor, centralprocessing unit (CPU), graphics processing unit (GPU), etc. Moreparticularly, image processor 311 may be a complex instruction setcomputing (CISC) microprocessor, reduced instruction set computing(RISC) microprocessor, very long instruction Word (VLIW) microprocessor,a processor implementing other instruction sets, or processorsimplementing a combination of instruction sets. In various embodiments,image processor 311 may also be one or more special-purpose processingdevices, such as an application specific integrated circuit (ASIC), afield programmable gate array (FPGA), a digital signal processor (DSP),a System on a Chip (SoC), etc. In some aspects, image processor 311 maybe communicatively coupled to memory 312 and configured to execute thecomputer-executable instructions stored thereon.

Memory 312 may be a non-transitory computer-readable medium, such as aread-only memory (ROM), a random access memory (RAM), a phase-changerandom access memory (PRAM), a static random access memory (SRAM), adynamic random access memory (DRAM), an electrically erasableprogrammable read-only memory (EEPROM), other types of random accessmemories, a flash disk or other forms of flash memory, a cache, aregister, a static memory, a compact disc read-only memory (CD-ROM), adigital versatile disc (DVD) or other optical storage, a cassette tapeor other magnetic storage devices, or any other non-transitory mediumthat may be used to store information including images, data, orcomputer executable instructions capable of being accessed by aprocessor, or any other type of computer device, etc.

In some embodiments, memory 312 may store computer-executableinstructions, such as one or more image segmentation programs 321, aswell as data used or generated while executing the computer programs,such as image data 322. Image processor 311 may execute programs 321 tosegment the subject images using ABAS methods. For example, imagesegmentation programs 321 may be programs to estimate structure labelsfor the subject images. In some embodiments, programs 321 may performvarious functions, such as atlas registration and/or selection, trainingof a structure classifier using the registered and/or selected atlas,and segmentation of subject images using the trained structureclassifier.

Image processor 311 may also send and/or receive image data 322 frommemory 312. Image processor 311 may communicate with database 320 toread image data 322 into memory 312 or store image data 322 from memory312 to image database 320. In some embodiments, image data 322 mayinclude subject images acquired MRI images, CT images, PET images,ultrasound images, and computer generated synthetic images, etc. Imagedata 322 may further include atlas images that are pre-collected andprocessed, and stored in database 320. Image processor 311 may alsogenerate intermediate data such as registered atlas images, structurelabels, parameters of the any classifier model, and send them to memory312.

Storage device 316 may be an additional storage available to store dataand/or computer programs associated with image processing tasksperformed by image processor 311. In some embodiments, storage device316 may include a machine-readable storage medium. While themachine-readable storage medium in an embodiment may be a single medium,the term “machine-readable storage medium” should be taken to include asingle medium or multiple media (e.g., a centralized or distributeddatabase, and/or associated caches and servers) that store the one ormore sets of computer executable instructions or data. The term“machine-readable storage medium” shall also be taken to include anynon-transitory medium that is capable of storing or encoding a set ofinstructions for execution by the machine and that cause the machine toperform any one or more of the methodologies of the present disclosure.The term “machine-readable storage medium” shall accordingly be taken toinclude, but not be limited to, solid-state memories, optical andmagnetic media.

Input/output interface 313 may be configured to allow data to bereceived and/or transmitted by medical image processing device 310,consistent with the disclosed embodiments. Input/output interface 313may include one or more digital and/or analog communication devices thatallow medical image processing device 310 to communicate with a user orother machines and devices. For example, input/output interface 313 mayinclude a keyboard and a mouse for the user to provide input intomedical image processing device 310. In an embodiment, the input/outputinterface 313 may be a cellular device such as a mobile phone, a tabletdevice such as an iPad, or any other handheld electronic device that iscapable of interfacing with the medical image processing device 310.Such a tablet or mobile device includes a display for depicting medicalimage data and medical images. Further, such tablet or mobile device canbe configured with a touch-screen display to manipulate the data andimages.

Network interface 314 may be configured to enable medical imageprocessing device 310 to communicate over a network, such as network360, consistent with disclosed embodiments. In some embodiments, networkinterface 360 may include at least one of a network adaptor, a cableconnector, a serial connector, a USB connector, a parallel connector, ahigh-speed data transmission adaptor, such as fiber, USB 3.0,thunderbolt, and the like, a wireless network adaptor, such as a Wi-Fiadaptor, a telecommunication (3G, 4G/LTE and the like) adaptor, etc.Medical image processing device 500 may be connected to network 460through network interface 314.

Image display 315 may be any display device suitable for displaying theimages, consistent with disclosed embodiments. For example, imagedisplay 315 may be an LCD display, CRT display, LED display, organiclight-emitting diode, organic light emitting transistor, field emissiondisplay, quantum dot or liquid crystal displays, MEMS display, Ferroliquid display, thick-film dielectric electroluminescent display, bendydisplays, foldable displays, haptic touchscreens, virtual realitydisplays, 3D pixel displays, virtual retina display, holographicdisplay, laser phosphor display and the like.

Network 360 may be configured to provide communications between thecomponents of FIG. 3. For example, network 360 may be any type ofnetwork (including infrastructure) that provides communications,exchanges information, and/or facilitates the exchange of electronicinformation. In this regard, network 360 may include a wired connection,a wireless connection, a computer bus, a serial connection, a parallelconnection, an Ethernet connection, a local area network or a wide areanetwork, an internet connection, a satellite connection, or any othersuitable connection(s), including a connection to a cloud computingservice, or any combination thereof that enables components of imagesegmentation system 300 to send and to receive information among eachother in any format and under any communications protocol.

It is contemplated that FIG. 3 illustrates only an exemplary arrangementof image segmentation system 300. In some embodiments, additionalcomponents may be added, and/or the depicted components may be combined,divided, modified, or removed. Further, in some aspects, at least onecomponent of image segmentation system 300 may be geographically remotefrom the remaining components, and may communicate with the remainingcomponents through network 360.

FIG. 4 is a flow chart illustrating an exemplary image segmentationmethod 400, consistent with the disclosure. In some embodiments, method400 may be performed by components of image segmentation system 300,such as medical image processing device 310, to segment one or moresubject images. Although segmentation of only one structure of interestis described as an example, it is contemplated that method 400 can beapplied to segment a group of structures of interest at the same time,such as bladder, prostate, and rectum, which are spatially adjacent andhighly correlated. Various machine learning methods, such as RandomForest (RF) method, can naturally handle segmentation of multiplestructures at the same time. A multi-structure classifier model may bebeneficial when the multiple structures are spatially adjacent and thushighly correlated.

As shown in FIG. 4, at 402, medical image processing device 310 mayreceive image data 322, including a subject image and one or moreatlases, from database 120. In the embodiment shown in FIG. 4, Natlases, Atlas #1, Atlas #2, . . . , and Atlas #N, are used. Althoughusing multiple atlases may be beneficial to improve segmentationaccuracy, methods consistent with the disclosure can be applied to thescenario where only one atlas is used. Each of the atlases includes anatlas image and corresponding structure delineations (annotations).

At 404, an image registration is performed to register the atlas imageswith the subject image. In some embodiments, the registration processmay include mapping the image points of each atlas image to the subjectimage points. In some alternative embodiments, the registration processmay include mapping both the atlas images and the subject image to areference image. In these embodiments, the reference image can be, forexample, an average atlas image or a common template image. As such, theatlas images are “indirectly” mapped to the subject image. Various imageregistration methods can be used, such as one or a combination of any ofa linear registration, an object-driven “poly-smooth” non-linearregistration, or a shape-constrained dense deformable registration. Byperforming the image registration, an image transformation from theatlas image to the reference image is calculated for each atlas.

At 406, the delineations (e.g., structure labels) of each atlas aremapped to the space of the reference image using the corresponding imagetransformation for the atlas. The mapped structure labels representindependent classification data, i.e., independent segmentation results,of the subject image from the corresponding atlas.

As described above, the processes in 404 and 406 result in mapped atlasimages, also referred to as “registered atlas images,” and mappedstructure labels, also referred to as “registered structure labels.” Amapped atlas image and corresponding mapped structure labels constitutea mapped atlas, also referred to as a “registered atlas,” which can thenbe used to train a structure classifier for classifying the subjectimage, as will be described later.

Referring again to FIG. 4, at 408, a label fusion is performed tocombine the segmentation results, e.g., label maps, from differentatlases to obtain a consensus ABAS segmentation for the structure ofinterest. Various label fusion methods, such as majority voting andsimultaneous truth and performance level estimation (STAPLE), can beemployed to combine the mapped structure labels of the different atlasesinto a consensus ABAS structure label estimation, which includesestimated structure labels, also referred to as ABAS labels, for thesubject image points. For example, with a majority voting technique, ateach subject image point, each mapped structure label casts a voteregarding whether the corresponding subject image point belongs to thestructure of interest or not. The final label of the subject image pointcan be determined as the one label that has the most votes. For example,in a binary case where the label value is either 1 (for being inside thestructure of interest) or 0 (for being outside of the structure ofinterest), the majority voting can be calculated by taking the averageof all the labels at the corresponding subject image point and thenassigning the subject image point as being inside the structure ofinterest or outside of the structure of interest depending on whetherthe average is higher or lower than a threshold value, such as 0.5. Asanother example, in the STAPLE method, optimal, non-equal weights can beassigned to different atlases based on some intelligent estimation ofthe performance or accuracy of each individual atlas. When there aremultiple structures of interest, each structure label may indicate whichstructure the subject image point belongs to, or alternatively, includea set of probability values indicating the probability of the subjectimage point to belong to the respective structures. The structure labelscan also be fused using the same methods described above.

At 410, the structure classifier is trained using one or more of themapped atlases. In some embodiments, all of the mapped atlases outputfrom the processes of 404 and 406 are used for training the structureclassifier. In some embodiments, only one or some of the mapped atlasesare selected for the training purpose. In these embodiments, an atlasselection can be performed to obtain a subset of mapped atlases that aresuitable for the subject patient. Various criteria can be used for theatlas selection. For example, an image similarity between each mappedatlas image and the subject image (in the scenario where the subjectimage is the reference image) or the mapped subject image (in thescenario where another image is used as the reference image) can beevaluated and the mapped atlases can be ranked based on the imagesimilarities. The image similarity can be a global similarity, i.e., asimilarity between the entire mapped atlas image and the entire subjectimage/mapped subject image, or a local image similarity, i.e., asimilarity between the structure of interest in the mapped atlas imageand the structure of interest in the subject image/mapped subject image.For example, the similarity can be calculated as a correlation betweenthe two images, or corresponding portions of the two images. In someembodiments, mapped atlases having a high degree of confidence as to theaccuracy of their corresponding classification data are used for thetraining purpose. In some other embodiments, atlases may be rankedhigher if they are associated with the same patient, from which thesubject image is acquired.

Mapped atlases with higher ranks are used for the training. Detailedprocess for training the structure classifier will be described laterwith reference to FIG. 5. After the structure classifier is trained, thestructure classifier can be used to determine structure labels for inputimage points based on corresponding attributes of the image points.

As shown in FIG. 4, at 412, the ABAS structure label estimation obtainedfrom the label fusion is used to identify what subject image points areto be further analyzed using the trained structure classifier. Varioustechniques can be used to select a subset of subject image points forfurther analysis. For example, criteria can be defined for assessingwhether the ABAS label for a subject image point is ambiguous, and thensubject image points for which the ABAS labels are ambiguous can beincluded in the subset for further analysis.

As an example of such ambiguity criteria, subject image points for whichthere is a disagreement regarding classification among the various ABASlabels obtained from different atlas registrations are included in thesubset for further analysis. In some embodiments, the ABAS label foreach subject image point is accompanied with an estimated accuracy(e.g., a probability value). In these scenarios, the estimated accuracycan be used to determine whether the associated subject image point isambiguous. For example, subject image points having an accuracy of,e.g., 50%, or below are included in the subset for further analysis. Asanother example, a proximity around the border of the structure ofinterest can be defined and subject image points within the proximitycan be included in the subset for further analysis.

In some embodiments, attributes, also known as image features, that areto be used by the trained structure classifier are also computed at 412.The attributes can be the same types of attributes used for training thestructure classifier, as will be described in more detail below inconnection with FIG. 5. Various methods may be used to compute theattributes, including using machine learning models such asconvolutional neural network models.

At 414, the subject image points in the subset for further analysis areapplied to the trained structure classifier, which will provide anotherset of structure labels for those image points in the subset. This setof structure labels estimated by the classifier may or may not agreewith the ABAS structural labels generate in 408.

At 416, the ABAS structure labels and the classifier structure labelsare combined to generate final structure labels, which represents thefinal segmentation result for the subject image. Various techniques canbe used to combine the labels. For example, majority voting between theABAS structure label estimation and the classifier structure labelestimation can be employed. As another example, if the trained structureclassifier produces a hard decision, the results from the structureclassifier can be taken as another label map and a label fusion betweenthe label map resulting from the ABAS and the label map generated by thestructure classifier can be performed. As yet another example, if ABASand the structure classifier provide estimated probabilities, P_(L) andP_(C), respectively, as structure labeling results, a final structureprobability P for a subject image point can be calculated as a weightedaverage of the estimate probabilities from the two techniques:P=w _(L) P _(L) +w _(C) P _(C), where w _(L) +w _(C)=1.The weights w_(L) and w_(C) can be equal to or different from eachother. The weights can be manually set or automatically determined basedon a training procedure such as cross-validation. Once the finalstructure probability P for a subject image point is calculated, whetherthe subject image point belongs to the structure of interest can bedetermined by determining whether the final structure probability P ishigher than a threshold, such as 0.5.

FIG. 5 is a flow chart schematically showing an exemplary method 500 fortraining the structure classifier using one or more of the mappedatlases. A mapped atlas used for the training purpose is also referredto as a training atlas. According to the disclosure, a machine learningalgorithm may be applied to data from the training atlases (the atlasimages and their associated classification data) to produce the trainedstructure classifier. According to the disclosure, the structureclassifier can be trained before the label fusion (method 400 at 408) isperformed, after the label fusion is performed but before the subjectimage points for further analysis are identified (method 400 at 412),after the subject image points for further analysis are identified, orduring either of these two processes.

As shown in FIG. 5, at 502, a plurality of training samples can beselected from the mapped atlas image of each training atlas. Eachtraining sample can correspond to a single image point or a group ofimage points (such a group of image points is also referred to as asuper image point). According to the disclosure, the training samplesfrom a mapped atlas image can include all or a portion of the imagepoints on the mapped atlas image. When only a portion of the imagepoints are used for training, a sample selection can be performed todetermine what image points are used. For example, the training samplescan be selected fully randomly over the entire mapped atlas image, or beselected from a region within a certain distance to the border of thestructure of interest. Examples of selecting training samples from sucha region are described in U.S. Pat. No. 9,122,950. As another example,the sample selection can be guided by the registration results such thatmore samples can be selected from an ambiguous region, i.e., the regionwhere structure labels from different mapped atlases do not completelyagree with each other or the disagreement is larger than a certain level(for example, three or more out of ten mapped atlases have a differentdetermination than the other mapped atlases).

At 504, a plurality of attributes (or features) can be computed for theselected training samples. These attributes are to be used by themachine learning algorithm as part of its classification task. Whenbeing applied to classify an image point of a subject image, the trainedstructure classifier will make decisions about the structure label ofthe image point based on computed attributes associated with the imagepoint. One or more types of attributes can be computed for each trainingsample. Various types of attributes can be used, such as, for example,image intensity value, image location, image gradient and gradientmagnitude, eigen-values of a Hessian matrix of the image, image texturemeasures such as energy, entropy, contrast, homogeneity, and correlationof local co-occurrence matrix, local image patches of varying sizes, asdescribed in more detail in U.S. Pat. No. 9,122,950. Alternatively,attributes or features may also be automatically and adaptively computedusing machine learning models. For example, a convolutional neuralnetwork model may be trained to extract relevant features from sampleimages, and the pre-trained model can be applied to the training samplesto produce attributes. A convolutional neural network typically includesseveral convolution layers, among other layers, that produce featuremaps of various sizes. The feature maps contain generic featurescharacterizing the input image (or a selected portion of the inputimage), and thus can be used as features in the structure classifier tofurther improve classification results. Features from variousconvolution layers (e.g., top layers, middle layers, lower layers), or aselection of these layers, may be used. In some embodiments, computationof attributes can be omitted if the training atlases already include theattributes for the atlas image points that are to be used by the machinelearning algorithm.

At 506, the collected training samples and the computed attributes areapplied to the machine learning algorithm to produce the trainedstructure classifier. The machine learning algorithm can be a supervisedlearning algorithm, which seeks to infer a prediction model given a setof training data. For example, the machine learning algorithm fortraining the structure classifier can be the random forests (RF) machinelearning algorithm, which can naturally handle multiple classes, i.e.,one classifier to classify several structures. The output of an RFclassifier can be a probability estimation of which class the input databelongs to, i.e., which structure the corresponding image point belongsto. The RF algorithm is described in more detail in U.S. Pat. No.9,122,950. After the trained structure classifier is obtained, it can bestored and applied during later use in connection with auto-segmentationof the subject image, such as used in 414 of method 400.

FIGS. 6A and 6B show comparisons between the structure border estimatedby the STAPLE method (dashed black line) and the structure borderestimated by the online learning-enhanced ABAS method consistent withthe disclosure (solid white line). The subject images shown in FIGS. 6Aand 6B are images of the liver and the right kidney of a same subjectpatient. As shown in both FIGS. 6A and 6B, the online learning-enhancedABAS method produces more accurate borders than the STAPLE method. Inparticular, as shown in FIG. 6B, the STAPLE method misses the bottomportion of the kidney, whereas the estimated border from the onlinelearning-enhanced ABAS method better conforms to the actual border ofthe kidney.

The above detailed description includes references to the accompanyingdrawings, which form a part of the detailed description. The drawingsshow, by way of illustration, specific embodiments in which theinvention can be practiced. These embodiments are also referred toherein as “examples.” Such examples can include elements in addition tothose shown or described. However, examples in which only those elementsshown or described are provided. Moreover, any combination orpermutation of those elements shown or described (or one or more aspectsthereof), either with respect to a particular example (or one or moreaspects thereof), or with respect to other examples (or one or moreaspects thereof) shown or described herein are within the scope of thepresent disclosure.

In the event of inconsistent usages between this document and anydocuments so incorporated by reference, the usage in this documentcontrols.

In this document, the terms “a” or “an” are used, as is common in patentdocuments, to include one or more than one, independent of any otherinstances or usages of “at least one” or “one or more.” In thisdocument, the term “or” is used to refer to a nonexclusive or, such that“A or B” includes “A but not B,” “B but not A,” and “A and B,” unlessotherwise indicated. In this document, the terms “including” and “inwhich” are used as the plain-English equivalents of the respective terms“comprising” and “wherein.” Also, in the following claims, the terms“including” and “comprising” are open-ended, that is, a system, device,article, composition, formulation, or process that includes elements inaddition to those listed after such a term in a claim are still deemedto fall within the scope of that claim. Moreover, in the followingclaims, the terms “first,” “second,” and “third,” etc. are used merelyas labels, and are not intended to impose numerical requirements ontheir objects.

A machine or computer-readable storage medium may include one or morenon-transitory media (e.g., a centralized or distributed database,and/or associated caches and servers). Such a machine orcomputer-readable storage medium may store computer-executableinstructions or data that may cause a machine to perform the functionsor operations described. Such a machine or computer-readable storagemedium may include any mechanism that stores information in a formaccessible by a machine (e.g., computing device, electronic system, andthe like), such as recordable/non-recordable medium (e.g., read onlymemory (ROM), random access memory (RAM), magnetic disk storage medium,optical storage medium, flash memory devices, and the like). Forexample, the term “machine-readable storage medium” or“computer-readable storage medium” shall accordingly be taken toinclude, but not be limited to, solid-state memories, optical, andmagnetic medium.

The above description is intended to be illustrative, and notrestrictive. For example, the above-described examples (or one or moreaspects thereof) may be used in combination with each other. Otherembodiments can be used, such as by one of ordinary skill in the artupon reviewing the above description. The Abstract is provided to complywith 37 C.F.R. § 1.72(b), to allow the reader to quickly ascertain thenature of the technical disclosure. It is submitted with theunderstanding that it will not be used to interpret or limit the scopeor meaning of the claims. Also, in the above Detailed Description,various features may be grouped together to streamline the disclosure.This should not be interpreted as intending that an unclaimed disclosedfeature is essential to any claim. Rather, inventive subject matter maylie in less than all features of a particular disclosed embodiment.Thus, the following claims are hereby incorporated into the DetailedDescription as examples or embodiments, with each claim standing on itsown as a separate embodiment, and it is contemplated that suchembodiments can be combined with each other in various combinations orpermutations. The scope of the invention should be determined withreference to the appended claims, along with the full scope ofequivalents to which such claims are entitled.

What is claimed is:
 1. An image segmentation method, comprising:receiving a plurality of atlases and a subject image, each atlasincluding an atlas image showing a structure of interest and associatedstructure delineations, the subject image being acquired by an imageacquisition device and showing the structure of interest; calculating,by an image processor, mapped atlases by registering the respectiveatlases to the subject image; determining, by the image processor, afirst structure label map for the subject image based on the mappedatlases; determining that a first structure label of a first set of themapped atlases for a given region of the subject image is different froma second structure label of a different second set of the mapped atlasesfor the given region; in response to determining that the firststructure label is different from the second structure label, selectingtraining samples corresponding to the given region for a subset of themapped atlases comprising at least one of the plurality of atlasesregistered to the subject image; training, by the image processor, astructure classifier using the selected training samples correspondingto the given region; determining, by the image processor, a secondstructure label map for the subject image by applying the trainedstructure classifier to one or more subject image points in the subjectimage; and combining, by the image processor, the first label map andthe second label map to generate a third label map representative of thestructure of interest.
 2. The method of claim 1, wherein calculating themapped atlases includes: mapping the atlas image in each atlas to thesubject image; calculating a registration transformation for each atlasbased on the mapping; and calculating mapped structure delineations foreach atlas by applying the registration transformation to the structuredelineations of the atlas.
 3. The method of claim 2, wherein determiningthe first structure label map includes: determining atlas-basedauto-segmentation (ABAS) structure label maps corresponding to theatlases based on the respective mapped structure delineations; anddetermining the first structure label map by fusing the ABAS structurelabel maps.
 4. The method of claim 3, wherein fusing the ABAS structurelabel is according to at least one of a majority voting method or asimultaneous truth and performance level estimation (STAPLE) method. 5.The method of claim 1, wherein registering the respective atlases to thesubject image includes mapping each atlas image and the subject image toa common reference image.
 6. The method of claim 5, wherein thereference image is an average atlas image obtained by averaging theatlas images.
 7. The method of claim 1, further comprising: selectingthe subset of the mapped atlases based on a selection criterion.
 8. Themethod of claim 7, wherein selecting the subset of the mapped atlasesincludes: determining an image similarity between each :napped atlasimage and the subject image; ranking the mapped atlases based on theimage similarities of the respective mapped atlas images; and selectingthe subset of the mapped atlases based on the ranking.
 9. The method ofclaim 8, wherein determining the image similarity includes determining aglobal similarity indicating how the corresponding mapped atlas image asa whole correlates with the subject image as a whole, or determining alocal similarity representing how the structure of interest in thecorresponding mapped atlas image correlates with the structure ofinterest in the subject image.
 10. The method of claim 1, wherein themachine learning algorithm to the one or more mapped atlases includesapplying a random forest algorithm to the one or more mapped atlases toobtain a random forest model.
 11. The method of claim 1, furthercomprising: selecting a plurality of training samples from each of theone or more mapped atlases; and training the structure classifier usingthe plurality of training samples.
 12. The method of claim 11, whereindetermining that the first structure label of the first set of themapped atlases is different from the second structure label of thesecond set of the mapped atlases comprises determining that a number ofmapped atlases in the first set exceeds a specified amount such that adisagreement between structure labels among the mapped atlases for thegiven region is larger than a certain level.
 13. The method of claim 11,further comprising: computing attributes for the training samples,wherein training the structure classifier uses the attributes.
 14. Themethod of claim 13, wherein the attributes are computed using apre-trained convolutional neural network.
 15. The method of claim 1,further comprising: selecting the one or more subject image points fromthe subject image based on the first structure label map.
 16. The methodof claim 15, wherein the one or more selected subject image pointscorrespond to structural labels indicative of segmentation ambiguity.17. An image segmentation apparatus, comprising: a memory configured toreceive and store a plurality of atlases and a subject image, each atlasincluding an atlas image showing a structure of interest and associatedstructure delineations, the subject image being acquired by an imageacquisition device and showing the structure of interest; and an imageprocessor coupled to the memory and configured to: calculate mappedatlases by registering the respective atlases to the subject image;determine a first structure label map for the subject image based on themapped atlases; determine that a first structure label of a first set ofthe mapped atlases for a given region of the subject image is differentfrom a second structure label of a different second set of the mappedatlases for the given region; in response to determining that the firststructure label is different from the second structure label, selecttraining samples corresponding to the given region for a subset of themapped atlases comprising at least one of the plurality of atlasesregistered to the subject image; train a structure classifier using theselected training samples corresponding to the given region; determine asecond structure label map for the subject image by applying the trainedstructure classifier to one or more subject image points in the subjectimage; and combine the first label map and the second label map togenerate a third label map representative of the structure of interest.18. The image segmentation apparatus of claim 17, wherein the imageprocessor is further configured to: select the subset of the mappedatlases based on image similarities between the respective mapped atlasimages and the subject image.
 19. The image segmentation apparatus ofclaim 17, wherein the image processor is further configured to: selectthe one or more subject image points from the subject image based on thefirst structure label map.
 20. A non-transitory computer-readablestorage medium storing instructions that, when executed by an imageprocessor, cause the processor to perform an image segmentation method,comprising: receiving a plurality of atlases and a subject image, eachatlas including an atlas image showing a structure of interest andassociated structure delineations, the subject image being acquired byan image acquisition device and showing the structure of interest;calculating mapped atlases by registering the respective atlases to thesubject image; determining a first structure label map for the subjectimage based on the mapped atlases; determining that a first structurelabel of a first set of the mapped atlases for a given region of thesubject image is different from a second structure label of a differentsecond set of the mapped atlases for the given region; in response todetermining that the first structure label is different from the secondstructure label, selecting training samples corresponding to the givenregion for a subset of the mapped atlases comprising at least one of theplurality of atlases registered to the subject image; training, by theimage processor, a structure classifier using the selected trainingsamples corresponding to the given region; determining a secondstructure label map for the subject image by applying the trainedstructure classifier to one or more subject image points in the subjectimage; and combining the first label map and the second label map togenerate a third label map representative of the structure of interest.